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^N) ' Abstract. Quantitative properties of stochastic systems are usually spec- 

K^_^. ified in logics that allow one to compare the measure of executions sat- 

frt ' isfying certain temporal properties with thresholds. The model checking 

^H ^ problem for stochastic systems with respect to such logics is typically 

^H ■ solved by a numerical approach [24I5I28I17I16I3) that iteratively com- 

QQ ' putes (or approximates) the exact measure of paths satisfying relevant 

, subformulas; the algorithms themselves depend on the class of systems 

^~i' being analyzed as well as the logic used for specifying the properties. 

Q^ ' Another approach to solve the model checking problem is to simulate 

l_J , the system for finitely many runs, and use hypothesis testing to infer 

^ ' whether the samples provide a statistical evidence for the satisfaction 

^ I or violation of the specification. In this short paper, we survey the sta- 

' ' . tistical approach, and outline its main advantages in terms of efficiency, 

' uniformity, and simplicity. 

^ I 1 Introduction and Context 

m ■ 

'""J I Quantitative properties of stochastic systems are usually specified in logics that 

l/^ ' allow one to compare the measure of executions satisfying certain temporal prop- 

^D I erties with thresholds. The model checking problem for stochastic systems with 

respect to such logics is typically solved by a numerical approach that iteratively 
computes (or approximates) the exact measure of paths satisfying relevant sub- 
formulas. The algorithm for computing such measures depends on the class of 
stochastic systems being considered as well as the logics used for specifying the 
5J] ' correctness properties. Model checking algorithms for a variety of contexts have 

5^ I been discovered [1I9I5J and there are mature tools (see e.g. J18I4] ) that have been 

used to analyze a variety of systems in practice. 

Despite the great strides made by numerical model checking algorithms, there 
are many challenges. Numerical algorithms work only for special systems that 
have certain structural properties. Further the algorithms require a lot of time 
and space, and thus scaling to large systems is a challenge. Finally, the logics 
for which model checking algorithms exist are extensions of classical temporal 
logics, which are often not the most popular among engineers. 

Another approach to verify quantitative properties of stochastic systems is 
to simulate the system for finitely many runs, and use hypothesis testing to 
infer whether the samples provide a statistical evidence for the satisfaction or 
violation of the specification [34] . The crux of this approach is that since sample 
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runs of a stochastic system are drawn according to the distribution defined by 
the system, they can be used to get estimates of the probabihty measure on 
executions. Starting from time-bounded PCTL properties [34] , the technique has 
been extended to handle properties with unbounded until operators [26j , as well 
as to black-box systems J25I30J . Tools based on this idea have been built |27l32j , 
and they have been used to analyze many systems. 

This approach enjoys many advantages. First, these algorithms only require 
that the system be executable (or rather, sample executions be drawn according 
to the measure space defined by the system). Thus, it can be applied to larger 
class of systems than numerical model checking algorithms including black-box 
systems and infinite state systems. Second the approach can be generalized to a 
larger class of properties, including Fourier transform based logics. Finally, the 
algorithm is easily parallelizable, which can help scale to large systems. However, 
the statistical approach also has some disadvantages when compared with the 
numerical approach. First, it only provides probabilistic guarantees about the 
correctness of the algorithms answer. Next, the sample size grows very large 
if the model checker's answer is required to be highly accurate. Finally, the 
statistical approach only works for purely probabilistic systems, i.e., those that 
do not have any nondetcrminism. Furthermore, since statistical tests are used to 
determine the correctness of a system, the approach only works for systems that 
"robustly" satisfy a given property, i.e., the actual measure of paths satisfying a 
given subformula, is bounded away from the thresholds to which it is compared 
in the specification. 

In this short paper, we will overview some of existing statistical model check- 
ing algorithms and discuss their efficiency. We will present the hypothesis testing 
algorithms that are at the heart of most of statistical algorithms and show how 
to uniformly analyze a large class of systems and properties. We will also discuss 
case studies. 

Most of these results are taken from Haakan Youncs PhD Thesis [31] . 



2 What do we want to do? 

We consider a stochastic system S and a property (j). An execution of 5 is a 
possibly infinite sequence of states of 5. Our objective is to solve the probabilistic 
model checking problem, i.e., to decide whether S satisfies (f> with a probability 
greater or equal to a certain threshold 9. The latter is denoted S (= P>g{<f>), 
where P is called a probabilistic operator. This paper will overview solutions to 
this problem. These solutions depend on the nature of S and (p. We consider 
three cases. 

1. We first assume that 5 is a white-box system, i.e., that one can generate 
as much executions of the system as we want. We also assume that (j) does 
not contain probabilistic operators. In Section |31 we recall basic statistical 
algorithms that can be used to verify bounded properties (i.e., properties 
that can be verified on fixed-length execution) of white-box systems. 



2. In Sectional we discuss extensions to the full probabilistic computation tree 
logic [5] • There, we consider the case where (j) can also contain probabilistic 
operators and the case where it has to be verified on infinite executions. 

3. In Section [Sj we briefly discuss the verification of black-box systems, i.e. 
systems for which a part of the probability distribution is unknown. 

In addition, in Section [51 we will present various experiments that show that 
(1) statistical model checking algorithms are more efficient than numerical ones, 
and (2) statistical model checking algorithms can be applied to solve problems 
that are beyond the scope of numerical methods. Finally, Section [7] discusses the 
future of statistical model checking. 

Remark 1. The objective of the present tutorial is not to feed the reader with 
technical details, but rather to present the concepts of statistical model checking, 
and outline its main advantages in terms of efficiency, uniformity, and simplicity. 

Remark 2. There arc other techniques that allow to estimate the probability 
for S to satisfies (j>. Those techniques, which arc based on Montc-Carlo tech- 
niques, will not be presented in this paper. The interested reader is redirected 
to |13|15|19] for more details. 

3 Statistical Model Checking : The Beginning 

In this section, we overview several statistical model checking techniques. We 
assume that <S is a white-box system and that (/) is a bounded property. By 
bounded properties, we mean properties that can be defined on finite executions 
of the system. In general, the length of such executions has to be pre-computed. 
Let Bi be a discrete random variable with a Bernoulli distribution of param- 
eter p. Such a variable can only take 2 values and 1 with Pr[Bi = 1] = p 
and Pr[Bi = 0] = 1 — p. In our context, each variable Bi is associated with one 
simulation of the system. The outcome for Bi, denoted 6^, is 1 if the simulation 
satisfies (j) and otherwise. To make sure that the above approach works, one 
has to make sure that one can get the result of any experiment in a finite amount 
of time. In general, this means that we are considering bounded properties, i.e., 
properties that can be decided on finite executions. 

Remark 3. All the results presented in this section are well-known mathemati- 
cal results coming from the area of statistics. As we shall see, these results are 
sufficient to verify bounded properties of a large class of systems. As those prop- 
erties are enough in many practical applications, one could wonder whether the 
contribution of the computer scientist should not be at the practical level rather 
than at the theoretical one. 

Before going further one should answer one last question: "What is the class 
of m,odels that can be considered?" In fact, the answer is quite simple: any 
stochastic system on which one can define a probability space for the property 



under consideration. Hence, statistical model checking provides a uniform ap- 
proach for the verification of a wide range of stochastic models, including Markov 
Chains or Continuous Timed Markov Chains. In general, one does not make the 
hypothesis that the system has the Markovian propertju, except when work- 
ing with nested formulas (see Section |4|) . There is a big warning: the technique 
cannot be used to verify properties of models that combine both nondetermin- 
istic and stochastic aspects. Indeed, the simulation-based approach could not 
distinguish between the probability distributions that are sampled. 

3.1 Qualitative Answer using Statistical Model Checking 

The main approaches |31I25| proposed to answer the qualitative question are 
based on hypothesis testing. Let p = Pr[(j)), to determine whether p > 6, we 
can test H : p > 9 against K : p < 6. A test-based solution does not guarantee 
a correct result but it is possible to bound the probability of making an error. 
The strength (a,/3) of a test is determined by two parameters, a and /3, such 
that the probability of accepting K (respectively, H) when H (respectively, K) 
holds, called a Type-I error (respectively, a Type-II error ) is less or equal to a 
(respectively, /?). 

A test has ideal performance if the probability of the Type-I error (respec- 
tively, Type-II error) is exactly a (respectively, f3). However, these requirements 
make it impossible to ensure a low probability for both types of errors simulta- 
neously (see [3T] for details). A solution to this problem is to relax the test by 
working with an indifference region {pi,po) with pQ>pi {po —pi is the size of the 
region). In this context, we test the hypothesis Hq : p>Po against Hi : p<p\ 
instead of H against K. If the value of p is between pi and pq (the indifference 
region), then we say that the probability is sufficiently close to so that we 
are indifferent with respect to which of the two hypotheses K or H is accepted. 
The thresholds pq and pi are generally defined in term of the single threshold 
9, e.g., pi = 9 — 5 and po = 9 + 6. We now need to provide a test procedure 
that satisfies the requirements above. In the next two subsections, we recall two 
solutions proposed by Younes in j31|35j . 

Single Sampling Plan. To test Hq against Hi, we specify a constant c. If X]i=i ^i 
is larger than c, then Hq is accepted, else Hi is accepted. The difficult part in this 
approach is to find values for the pair (n,c), called a single sampling plan (SSP 
in short), such that the two error bounds a and /3 are respected. In practice, one 
tries to work with the smallest value of n possible so as to minimize the number 
of simulations performed. Clearly, this number has to be greater if a and /? are 
smaller but also if the size of the indifference region is smaller. This results in 
an optimization problem, which generally does not have a closed-form solution 
except for a few special cases [31] . In his thesis [31] , Younes proposes a binary 
search based algorithm that, given po,pi,a, (3, computes an approximation of 
the minimal value for c and n. 



^ i.e., that the probability to go to one state only depends on the state in where we 
are, not on the history of the execution. 



Remark 4- There are many variants of this algorithm. As an example, in |26] . 
Sen ct al. proposes to accept Hq if '=^ >P- Here, the difficulty is to find a 
value for n such that the error bounds arc valid. 

Sequential probability ratio test. The sample size for a single sampling plan is 
fixed in advance and independent of the observations that are made. However, 
taking those observations into account can increase the performance of the test. 
As an example, if we use a single plan (n, c) and the m > c first simulations 
satisfy the property, then we could (depending on the error bounds) accept 
Hq without observing the n ~ m other simulations. To overcome this problem, 
one can use the sequential probability ratio test (SPRT in short) proposed by 
Wald [S3]. The approach is briefiy described below. 

In SPRT, one has to choose two values A and B [A > B) (see bellow) 
that ensure that the strength of the test is respected. Let m be the number 
of observations that have been made so far. The test is based on the following 
quotient: 

Pim _ fr PrjB, = 6, I p = pi) _ pf" (1 - p,)^-d^ 

Pom fL\ Pr{Br =h\p = Po) p;;- (1 - po)™-<^™ ' ^ ' 

where dm = X]i"i^«- The idea behind the test is to accept Hq if 2ii^ > A, 
and Hi if ^^^ < B. The SPRT algorithm computes ^^^ for successive values 
of m until either Hq or Hi is satisfied; the algorithm terminates with probabil- 
ity l|29j. This has the advantage of minimizing the number of simulations. In 
his thesis [3T] , Younes proposed a logarithmic based algorithm SPRT that given 
Po,pi, a and /3 implements the sequential ratio testing procedure. 

Discussion. Computing ideal values Aid and Bid for A and B in order to make 
sure that we are working with a test of strength {a,/3) is a laborious procedure 
(see Section 3.4 of [29]). In his seminal paper [29], Wald showed that if one de- 
fines Aid>A = L-^El and Bid < B = ,, , , then we obtain a new test whose 
strength is (a', /3'), but such that a' + /S' < a + (3, meaning that either a'<a or 
/3' < p. In practice, we often find that both inequalities hold. This is illustrated 
with the following example taken from |31] . 

Example 1. Let po ~ 0.5, pi = 0.3, a ~ 0.2 and j3 = 0.1. If we use Aid>A = 
ii^ and B,d < B = -^4^)' then we are guaranteed that a'<0.222 and /3'<0.125. 
Through computer simulation (repruding the same experiments 10000 of time), 
we observe that q;'<0.175 and /3'<0.082. So the strength of the test is in reality 
better than the theoretical assumption. 



3.2 Some Generalities Regarding Efficiency 

The efficiency of the above algorithms is characterized by the number of sim- 
ulations needed to obtain an answer as well as the time it costs to compute a 



simulation. The latter often depends on the property under verification. Both 
numbers are expected numbers as they change from executions to executions and 
can only be estimated (see |31| for an explanation). However, some generalities 
are known. For example, it is known that, except for some situations, SPRT is 
always faster than SSP. When 9 = 1 (resp. 9 = 0) SPRT degenerates to SSP; 
it is not a problem since SSP is known to be optimal for such values. Observe 
that the time complexity of statistical model checking is independent from the 
state-space and that the space complexity is of the order of the state space. 
Also, the expected number of simulations for SSP is logarithmic with respect to 
a and /3 and linear with respect to the indifference region; for SPRT, the number 
depends on the probability distribution p. 

An interesting discussion on complexity of statistical model checking can be 
found in Section 5.4 of [21]. 

4 Statistical Model Checking: The Computer Science 
Contribution 

In the previous section, we have proposed statistical model checking algorithms 
for verifying bounded properties of white-box systems. In this section, we go one 
step further and consider three nontrivial extensions that are: 

1. The nested case, i.e., the case where </> can also contain probabilitistic oper- 
ators. Example: P>g-^{q => P>e2{4>2)) 

2. The unbounded case, i.e., the case where cannot be decide on a finite 
execution. Here we will restrict ourselves to the until property. Given two 
formulas (j)i and (/)2, the until operator ensures that (pi is true until 02 has 
been seen (and this must happen!). 

3. Boolean combinations of formulae, i.e., formulae of the form: P>g-^{(j)i) A 

We will only survey these results and give pointers to relevant papers. 

4.1 The Unbounded Case: Until 

We are now concerned with the verification of the until property. The property 
requires that a property 4>i remains valid until a property 4>2 has been seen. 
The problem is that we do not know a priori the moment when 02 will be 
satisfied. Hence, one has to reason on infinite execution. There are two works on 
this topics, one by Sen et al.|26j and one more recent work by Pekergin et al. 
[25] . We will not give details on these works, but the reader should know that 
Sen works by extending the model with extra probabilities, which makes the 
solution extremely slow. Pekergin uses the new technique of perfect simulation, 
which is (according to her experiments) not only faster than Sen's one, but also 
more general as it allows to study the steady-state operator for continuous timed 
Markov Chains. 



Remark 5. Contrary to the numerical results |28l3j The above results are not 
sufficient to verify properties of the form P>e(0), where (/) is a property expressed 
in Linear Temporal Logic |22j . Incomplete results regarding the verification of 
these properties with simulation-based techniques can be found in J15I13J . 

4.2 Nested Probability Operators 

We consider the problem of checking whether S satisfies (f> with a probability 
greater or equal to 9. However, contrary to what we have been doing so far, we 
will now assume that </> cannot be decided on a single execution, i.e., we will 
assume that (f> is of the form P>ei4>i- So, where is the difficulty? The difficulty 
is that (j) cannot be model checked on a single execution, but rather depends on 
another test. Hence, we have to provide a way to nest tests. In his thesis, Younes 
proposed the following theorem. 

Theorem 1. Let ip — P>g{(l)) be a property and assume that (j) can be verified 
with Type-I error a' and Type-II error /3' , then tp can be verified with Type-I 
error a and Type-II error /3, assuming that the indifference region is of size at 
least {{9 + S)il - a'), (1 - (1 - (6* - <5)))(1 - f3'). 

Hence one has to find a compromise between the size of the indifference 
region of the inner test and the outer one. There arc two interesting facts to 
know about nested operators: 

1. Even for bounded properties, the above result (and in fact, any result in 
the literature |26I31I30I32| ) only works for systems that have the Markovian 
property. 

2. In practice, the complexity (in term of number of sampling) becomes expo- 
nential in the number of tests. 

Remark 6. An interesting research direction would be to study the link with 
probabilistic testing pO] . 

4.3 Boolean Combinations 

We have to consider two operations, namely conjunction and negation (as it is 
known that any Boolean combination reduces to combinations of these two op- 
erators). We recall some results provided by Younes. We start with conjunction. 

Theorem 2. Let ip be the conjunction of n properties i/ii, . . . , 02- Assume that 
each (j)i can be decided with Type-I error Ui and Type-II error /3/ . Then (f> can be 
decided with Type-I error mini(ai) and Type-II error maxi((3i). 

The idea behind the proof of the theorem is that 

1. If we claim that the conjunction is not satisfied, this means that we have 
deduced that one of the operands is not. 



2. If we claim that the conjunction is satisfied, this means that we have con- 
cluded that all the operands are satisfied. As we may have made mistakes in 
each individual verification, we get inaxi(/3j;). 

For negation, the result is provided by the following theorem. 

Theorem 3. To verify a formula -itjj with Type-I error a and Type-II error jB, 
it is sufficient to verify tp with Type-I error /3 and Type-II error a. 



5 Black-box Systems: a note 

Black-box Systems is an interesting class of stochastic systems whose treatment 
is beyond the scope of numerical techniques. Roughly speaking, a black-box 
systems is simply a system whose probability distribution (i.e., set of behaviors) 
is not totally known and cannot be observed. Hence, one can view a black-box 
system as a finite set of executions prc-computed and for which no information 
is available. 

In the context of such systems, Type errors and indifference region cannot 
play a role. Indeed, those parameters influence the number of simulations that 
can be computed, but here the simulations are given and you cannot compute 
more! 

A solution to this problem is to conduct a SSP test, without indifference 
region (i.e., S set to 0) and assuming that the parameter n is fixed to the number 
of simulations that are given in advance. The difficulty is to chose the constant 
c in such a way that it becomes roughly equal to accept Hq or Hi ii 9 = p. In 
his thesis [31] and in [33| , Younes proposed a solution to the problem. He also 
shown that a previous solution proposed by Sen [25] is not correct. 

There are techniques to verify nested formulas over black-box systems. There 
exists no technique for the verification of unbounded properties. Hence there is 
still a lot of research to conduct in this area. 



6 Tools and Experiments 

At the origin, there are two tools that implements statistical model checking 
algorithms, namely ymer[5^ and Vesia[53. Vesto implements a variation of the 
single sampling plan algorithm. The choice of implementing the SSP algorithm is 
motivated by the fact that it is easier to parallelize as the number of simulations 
to perform is known in advance. However, in his thesis, Younes showed that 
sequential algorithms are also easily parallelizable. Ymer is limited to bounded 
properties while Vesta also incorporate the unbounded until. In [T7], the authors 
conducted several experiments that tend to show that (1) Ymer is faster than 
Vesta and (2) Vesta makes more false positive (selecting the bad hypothesis) than 
Ymer. Regarding the unbounded case, it seems that Vesta is not very efficient 
and can make a lot of false positive. Both Vesta and Ymer have been applied to 
huge case studies. A comparison of Ymer and Vesta with established tools such 



PRISMA can be found in [TT]. 

There are a wide range of situations for which the bounded case suffices. We 
have written a series of recent papers in where we propose apphcations of SSP 
and SPRT to interesting problems. In the rest of this section, we briefly recap 
the content of these papers. 



6.1 Verifying Circuits 



In |6I7) . we applied SPRT to verifying properties of mixed-signal circuits, i.e., 
circuits for which there is an interaction between analog (continuous) and digital 
(discrete) values. Our first contribution was to propose a version of stochastic 
discrete-time event systems that fits into the framework introduced by Younes 
with the additional advantage that it explicitly handles analog and digital sig- 
nals. We also introduced probabilistic signal linear temporal logic, a logic adapted 
to the specification of properties for mixed-signal circuits in the temporal domain 
and in the frequency domain. Our second contribution was the analysis oia. A—E 
modulator. A A—S modulator is an efficient Analog-to-Digital Converter circuit, 
i.e., a device that converts analog signals into digital signals. A common critical 
issue in this domain is the analysis of the stability of the internal state variables of 
the circuit. The concern is that the values that are stored by these variables can 
grow out of control until reaching a maximum value, at which point we say that 
the circuit saturates. Saturation is commonly assumed to compromise the qual- 
ity of the analog-to-digital conversion. In [TU] and [13] reachability techniques 
developed in the area of hybrid systems are used to analyze the stability of a 
third-order modulator. Their idea is to use such techniques to guarantee that for 
every input signal in a given range, the states of the system remain stable. While 
this reachability-based approach is sound, it has important drawbacks such as 
(1) signals with long duration cannot be practically analyzed, and (2) properties 
that are commonly specified in the frequency domain rather than in the time 
domain cannot be checked. Our results show that a simulation-based approach 
makes it possible to handle properties and signals that are beyond the scope of 
the reachability-based approach. As an example, in our experiments, we analyze 
discrete-time signals with 24000 sampling points in seconds, while the approach 
in [TO] takes hours to analyze signals with up to 31 sampling points. We are also 
able to provide insight into a question left open in [10] by observing that satu- 
ration does not always imply an improper signal conversion. This can be done 
by comparing the Fourier transform of each of the input analog signals with 
the Fourier transform of its corresponding digital signal. Such a property can 
easily be expressed in our logic and Model Checked with our simulation-based 
approach. We are unaware of other formal verification techniques that can solve 
this problem. Indeed, numerical techniques cannot reason on an execution at a 
time. 



6.2 Systems Biology 

In [5], we considered tlie verification of complex biological systems, we intro- 
duced a new tool, called BioLab, for formally reasoning about the behavior of 
stochastic dynamic models by integrating SPRT into the BioNetGen [11112] 
framework for rule-based modeling. We then used BioLab to verify the stochas- 
tic bistability of T-cell signalling. There are three more challenges in the systems 
biology area (the reader is invited to think about these problems and to check 
the existing literature): 

1. How to perform efficient simulations? 

2. How to take into account prior knowledge on the model? 

3. What are the logics dedicated to biologists than can be model checked with 
the statistical approach? 

Remark 7. In fact, statistical model checking techniques recently received a lot 
of attention in the area of systems biology. As an example, in 2009, Carnegie 
Mellon University was awarded a 10000000 grant for applying such techniques 
in the medical area. 



6.3 Heterogeneous applications 

In [2], we have proposed to apply statistical model checking techniques to the 
verification of heterogeneous applications. Systems integrating multiple heteroge- 
neous distributed applications communicating over a shared network are typical 
in various sensitive domains such as aeronautic or automotive embedded sys- 
tems. Verifying the correctness of a particular application inside such a system 
is known to be a challenging task, which is often beyond the scope of existing 
exhaustive validation techniques. 

In our paper, we proposed to exploit the structure of the system in order to 
increase the efficiency of the verification process. The idea is conceptually simple: 
instead of performing an analysis of the entire system, we proposed to analyze 
each application separately, but under some particular context /execution envi- 
ronment. This context is a stochastic abstraction that represents the interactions 
with other applications running within the system and sharing the computation 
and communication resources. The idea is to build such a context automatically 
by simulating the system and learning the probability distributions of key char- 
acteristics impacting the functionality of the given application. The abstraction 
can easily be analyzed with statistical model checking techniques. 

The overall contribution of our study is an application of the above method 
on an industrial case study, the heterogeneous communication system (HCS for 
short) deployed for cabin communication in a civil airplane. HCS is an heteroge- 
neous system providing entertainment services (ex : audio/video on passengers 
demand) as well as administrative services (ex: cabin illumination, control, audio 
announcements), which are implemented as distributed applications running in 
parallel, across various devices within the plane and communicating through a 



common Ethernet-based network. The HCS system has to guarantee stringent 
requirements, such as rehable data transmission, fault tolerance, timing and 
synchronization constraints. An important requirement is the accuracy of clock 
synchronization between different devices. This latter property states that the 
difference between the clocks of any two devices should be bounded by a small 
constant, which is provided by the user and depends on his needs (for example, 
to guarantee the fiability of another service). Hence, one must be capable to 
compute the smallest bound for which synchronization occurs and compare it 
with the bound expected by the user. Unfortunately, due to the large number 
of heterogeneous components that constitute the system, deriving such a bound 
manually from the textual specification is an unfeasible task. In this paper, we 
propose a formal approach that consists in building a formal model of the HCS, 
then we apply simulation-based algorithms to this model in order to deduce the 
smallest value of the bound for which synchronization occurs. We start with a 
fixed value of the bound and check whether synchronization occurs. If yes, then 
we make sure that this is the best one. If no, we restart the experiment with a 
new value. 

We have been able to derive precise bounds that guarantee proper synchro- 
nization for all the devices of the system. We also computed the probability to 
satisfy the property for smaller values of the bound, i.e., bounds that do not 
satisfy the synchronization property with probability 1. Being able to provide 
such an information is of clear importance, especially when the best bound is 
too high with respect to user's requirements. We have observed that the values 
we obtained strongly depend on the position of the device in the network. We 
also estimated the average and worst proportion of failures per simulation for 
bounds that are smaller than the one that guarantees synchronization. Checking 
this latter property has been made easy because statistical model checking al- 
lows us to reason on one execution at a time. Finally, we have also considered the 
influence of clock drift on the synchronisation results. The experiments highlight 
the generality of our technique, which could be applied to other versions of the 
HCS as well as to other heterogeneous applications. 

7 The Future of Statistical Model Checking 

There are various directions for future research in the statistical model checking 
area. Here is a list of possible topics. 

— Using efficient techniques for performing simulation is crucial to guarantee 
good performances for any statistical model checking algorithm. Unfortu- 
nately, the existing algorithms do not exploit efficient simulation techniques. 
It would thus be worth combining statistical model checking algorithms with 
such techniques (example : rare-event simulations, , ...). This is a huge im- 
plementation effort which also requires to define a methodology to select the 
good simulation technique to be applied. 

— Statistical model checking algorithms have not yet been applied to the veri- 
fication of multi-core systems, this area should be investigated. 



— Statistical model checking algorithms do not apply to systems that com- 
bine both stochastic and non deterministic aspects. Extending the results to 
such systems is however crucial to perform verification of security protocols, 
networking protocols, and performance protocols. 

— Statistical model checking algorithms reduce to decide between two hypoth- 
esis. In many areas, especially systems biology, we may have a prior knowl- 
edge on the probability to satisfy each hypothesis. Incorporating this prior 
knowledge in the verification process may considerably reduce the number 
of simulations needed for the algorithm to terminate. 

— Statistical model checking algorithms suppose that the property (j) can be 
checked on finite executions of the system. There are however many situ- 
ations where <f> cannot be checked in a finite amount of time. This is for 
example the case when is a long-run average or a steady state property. 
In systems biology, we are clearly interested in the study of such properties. 

— Verifying applications running within a huge heterogeneous system without 
is a challenging problem. In a recent work [5] , the authors have proposed 
a new simulation-based technique for solving such problem. The technique 
starts by performing simulations of the system in order to learn the context 
in where the application is used. Then, it creates a stochastic abstraction 
for the application, which takes the context information into account. Up 
to know, there is no automatic way to learn the context and derive the 
stochastic context. However, what we have observed so far is that it often 
takes the form of properties that cannot be expressed in classical temporal 
logic. Hence, statistical model checking may be our last resort to analyze the 
resulting abstraction. 

— Statistical model checking may help testers. In [5T], Cavalli et al. proposed 
to use statistical techniques for conformance testing of timed stochastic sys- 
tems. The technique should be automated. This could lead to new algorithms 
for verifying the so-called black-box systems. 
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